High-resolution neural recordings improve the accuracy of speech decoding.

Patients suffering from debilitating neurodegenerative diseases often lose the ability to communicate, detrimentally affecting their quality of life. One solution to restore communication is to decode signals directly from the brain to enable neural speech prostheses. However, decoding has been limited by coarse neural recordings which inadequately capture the rich spatio-temporal structure of human brain signals. To resolve this limitation, we performed high-resolution, micro-electrocorticographic (µECoG) neural recordings during intra-operative speech production. We obtained neural signals with 57× higher spatial resolution and 48% higher signal-to-noise ratio compared to macro-ECoG and SEEG. This increased signal quality improved decoding by 35% compared to standard intracranial signals. Accurate decoding was dependent on the high-spatial resolution of the neural interface. Non-linear decoding models designed to utilize enhanced spatio-temporal neural information produced better results than linear techniques. We show that high-density µECoG can enable high-quality speech decoding for future neural speech prostheses.

Flexible, high-resolution cortical arrays with large coverage capture microscale high-frequency oscillations in patients with epilepsy.

OBJECTIVE: Effective surgical treatment of drug-resistant epilepsy depends on accurate localization of the epileptogenic zone (EZ). High-frequency oscillations (HFOs) are potential biomarkers of the EZ. Previous research has shown that HFOs often occur within submillimeter areas of brain tissue and that the coarse spatial sampling of clinical intracranial electrode arrays may limit the accurate capture of HFO activity. In this study, we sought to characterize microscale HFO activity captured on thin, flexible microelectrocorticographic (µECoG) arrays, which provide high spatial resolution over large cortical surface areas. METHODS: We used novel liquid crystal polymer thin-film µECoG arrays (.76-1.72-mm intercontact spacing) to capture HFOs in eight intraoperative recordings from seven patients with epilepsy. We identified ripple (80-250 Hz) and fast ripple (250-600 Hz) HFOs using a common energy thresholding detection algorithm along with two stages of artifact rejection. We visualized microscale subregions of HFO activity using spatial maps of HFO rate, signal-to-noise ratio, and mean peak frequency. We quantified the spatial extent of HFO events by measuring covariance between detected HFOs and surrounding activity. We also compared HFO detection rates on microcontacts to simulated macrocontacts by spatially averaging data. RESULTS: We found visually delineable subregions of elevated HFO activity within each µECoG recording. Forty-seven percent of HFOs occurred on single 200-µm-diameter recording contacts, with minimal high-frequency activity on surrounding contacts. Other HFO events occurred across multiple contacts simultaneously, with covarying activity most often limited to a .95-mm radius. Through spatial averaging, we estimated that macrocontacts with 2-3-mm diameter would only capture 44% of the HFOs detected in our µECoG recordings. SIGNIFICANCE: These results demonstrate that thin-film microcontact surface arrays with both highresolution and large coverage accurately capture microscale HFO activity and may improve the utility of HFOs to localize the EZ for treatment of drug-resistant epilepsy.

Speech fine structure contains critical temporal cues to support speech segmentation.

Segmenting the continuous speech stream into units for further perceptual and linguistic analyses is fundamental to speech recognition. The speech amplitude envelope (SE) has long been considered a fundamental temporal cue for segmenting speech. Does the temporal fine structure (TFS), a significant part of speech signals often considered to contain primarily spectral information, contribute to speech segmentation? Using magnetoencephalography, we show that the TFS entrains cortical responses between 3 and 6 Hz and demonstrate, using mutual information analysis, that (i) the temporal information in the TFS can be reconstructed from a measure of frame-to-frame spectral change and correlates with the SE and (ii) that spectral resolution is key to the extraction of such temporal information. Furthermore, we show behavioural evidence that, when the SE is temporally distorted, the TFS provides cues for speech segmentation and aids speech recognition significantly. Our findings show that it is insufficient to investigate solely the SE to understand temporal speech segmentation, as the SE and the TFS derived from a band-filtering method convey comparable, if not inseparable, temporal information. We argue for a more synthetic view of speech segmentation - the auditory system groups speech signals coherently in both temporal and spectral domains.

Manipulating stored phonological input during verbal working memory.

Verbal working memory (vWM) involves storing and manipulating information in phonological sensory input. An influential theory of vWM proposes that manipulation is carried out by a central executive while storage is performed by two interacting systems: a phonological input buffer that captures sound-based information and an articulatory rehearsal system that controls speech motor output. Whether, when and how neural activity in the brain encodes these components remains unknown. Here we read out the contents of vWM from neural activity in human subjects as they manipulated stored speech sounds. As predicted, we identified storage systems that contained both phonological sensory and articulatory motor representations. Unexpectedly, however, we found that manipulation did not involve a single central executive but rather involved two systems with distinct contributions to successful manipulation. We propose, therefore, that multiple subsystems comprise the central executive needed to manipulate stored phonological input for articulatory motor output in vWM.

I see what you are saying.

The motor cortex in the brain tracks lip movements to help with speech perception.

A kiss is not a kiss: visually evoked neuromagnetic fields reveal differential sensitivities to brief presentations of kissing couples.

With a few exceptions, the literature on face recognition and its neural basis derives from the presentation of single faces. However, in many ecologically typical situations, we see more than one face, in different communicative contexts. One of the principal ways in which we interact using our faces is kissing. Although there is no obvious taxonomy of kissing, we kiss in various interpersonal situations (greeting, ceremony, sex), with different goals and partners. Here, we assess the visual cortical responses elicited by viewing different couples kissing with different intents. The study thus lies at the nexus of face recognition, action recognition, and social neuroscience. Magnetoencephalography data were recorded from nine participants in a passive viewing paradigm. We presented images of couples kissing, with the images differing along two dimensions, kiss type and couple type. We quantified event-related field amplitudes and latencies. In each participant, the canonical sequence of event-related fields was observed, including an M100, an M170, and a later M400 response. The earliest two responses were significantly modulated in latency (M100) or amplitude (M170) by the sex composition of the images (with male-male and female-female pairings yielding faster latency M100 and larger amplitude M170 responses). In contrast, kiss type showed no modulation of any brain response. The early cortical-evoked fields that we typically associate with the presentation and analysis of single faces are differentially sensitive to complex social and action information in face pairs that are kissing. The early responses, typically associated with perceptual analysis, exhibit a consistent grouping and suggest a high and rapid sensitivity to the composition of the kissing pairs.

Sensory-motor transformations for speech occur bilaterally.

Historically, the study of speech processing has emphasized a strong link between auditory perceptual input and motor production output. A kind of 'parity' is essential, as both perception- and production-based representations must form a unified interface to facilitate access to higher-order language processes such as syntax and semantics, believed to be computed in the dominant, typically left hemisphere. Although various theories have been proposed to unite perception and production, the underlying neural mechanisms are unclear. Early models of speech and language processing proposed that perceptual processing occurred in the left posterior superior temporal gyrus (Wernicke's area) and motor production processes occurred in the left inferior frontal gyrus (Broca's area). Sensory activity was proposed to link to production activity through connecting fibre tracts, forming the left lateralized speech sensory-motor system. Although recent evidence indicates that speech perception occurs bilaterally, prevailing models maintain that the speech sensory-motor system is left lateralized and facilitates the transformation from sensory-based auditory representations to motor-based production representations. However, evidence for the lateralized computation of sensory-motor speech transformations is indirect and primarily comes from stroke patients that have speech repetition deficits (conduction aphasia) and studies using covert speech and haemodynamic functional imaging. Whether the speech sensory-motor system is lateralized, like higher-order language processes, or bilateral, like speech perception, is controversial. Here we use direct neural recordings in subjects performing sensory-motor tasks involving overt speech production to show that sensory-motor transformations occur bilaterally. We demonstrate that electrodes over bilateral inferior frontal, inferior parietal, superior temporal, premotor and somatosensory cortices exhibit robust sensory-motor neural responses during both perception and production in an overt word-repetition task. Using a non-word transformation task, we show that bilateral sensory-motor responses can perform transformations between speech-perception- and speech-production-based representations. These results establish a bilateral sublexical speech sensory-motor system.

Visual input enhances selective speech envelope tracking in auditory cortex at a "cocktail party".

Our ability to selectively attend to one auditory signal amid competing input streams, epitomized by the "Cocktail Party" problem, continues to stimulate research from various approaches. How this demanding perceptual feat is achieved from a neural systems perspective remains unclear and controversial. It is well established that neural responses to attended stimuli are enhanced compared with responses to ignored ones, but responses to ignored stimuli are nonetheless highly significant, leading to interference in performance. We investigated whether congruent visual input of an attended speaker enhances cortical selectivity in auditory cortex, leading to diminished representation of ignored stimuli. We recorded magnetoencephalographic signals from human participants as they attended to segments of natural continuous speech. Using two complementary methods of quantifying the neural response to speech, we found that viewing a speaker's face enhances the capacity of auditory cortex to track the temporal speech envelope of that speaker. This mechanism was most effective in a Cocktail Party setting, promoting preferential tracking of the attended speaker, whereas without visual input no significant attentional modulation was observed. These neurophysiological results underscore the importance of visual input in resolving perceptual ambiguity in a noisy environment. Since visual cues in speech precede the associated auditory signals, they likely serve a predictive role in facilitating auditory processing of speech, perhaps by directing attentional resources to appropriate points in time when to-be-attended acoustic input is expected to arrive.

A mutual information analysis of neural coding of speech by low-frequency MEG phase information.

Recent work has implicated low-frequency (<20 Hz) neuronal phase information as important for both auditory (<10 Hz) and speech [theta (∼4-8 Hz)] perception. Activity on the timescale of theta corresponds linguistically to the average length of a syllable, suggesting that information within this range has consequences for segmentation of meaningful units of speech. Longer timescales that correspond to lower frequencies [delta (1-3 Hz)] also reflect important linguistic features-prosodic/suprasegmental-but it is unknown whether the patterns of activity in this range are similar to theta. We investigate low-frequency activity with magnetoencephalography (MEG) and mutual information (MI), an analysis that has not yet been applied to noninvasive electrophysiological recordings. We find that during speech perception each frequency subband examined [delta (1-3 Hz), theta(low) (3-5 Hz), theta(high) (5-7 Hz)] processes independent information from the speech stream. This contrasts with hypotheses that either delta and theta reflect their corresponding linguistic levels of analysis or each band is part of a single holistic onset response that tracks global acoustic transitions in the speech stream. Single-trial template-based classifier results further validate this finding: information from each subband can be used to classify individual sentences, and classifier results that utilize the combination of frequency bands provide better results than single bands alone. Our results suggest that during speech perception low-frequency phase of the MEG signal corresponds to neither abstract linguistic units nor holistic evoked potentials but rather tracks different aspects of the input signal. This study also validates a new method of analysis for noninvasive electrophysiological recordings that can be used to formally characterize information content of neural responses and interactions between these responses. Furthermore, it bridges results from different levels of neurophysiological study: small-scale multiunit recordings and local field potentials and macroscopic magneto/electrophysiological noninvasive recordings.